In the present day, implemented full text search and indexing mechanisms contribute to creating dictionaries that utilize a large amount of storage space. A typical dictionary contains term entries which are stored in a dictionary portion of an index. The larger the number of term entries, the slower the response when handling queries for search terms. When receiving a document, the document is processed for terms which are then added to the index of the dictionary. Typically, a term is added once to the dictionary and the term is referenced to all occurrences of the word in the documents the dictionary has received. The term is searchable, such that all occurrences of the word can be identified utilizing the index entry for the term.
Existing mechanisms for index entries reduce the size of the entries but also reduce recall for the entries. The reduction in recall is due to word filters removing a number of words or terms from a term being entered into the index. For example, existing mechanism can remove filler words, such as “a” or “the”, which the mechanism can deem unnecessary. While this reduces the size of the entry, it also reduces the accuracy of recall when receiving a search term.